USE 373 - reduce TIM bulk update logging#381
Merged
Merged
Conversation
Why these changes are being introduced: The opensearchpy library has a built-in DEBUG logger that logs the full API request + response for each request. This is not terribly helpful for debugging purposes, as in code we could log both the requets and/or response as needed. A good example are bulk updates where ALL documents we send for update are logged. For a job like DSpace or Alma, the amount of logs are equal to the amount of data we are sending. That said, DEBUG logging for the opensearch library is quite helpful for other reasons, so it's not ideal to completely quiet that library's DEBUG logs. Ideall we would only target this specific, private logging method. How this addresses that need: * Pinpoints and filters out the opensearchpy logging method, _log_request_response() Side effects of this change: * Dramatically reduced logs in DEBUG mode, while still getting helpful opensearch DEBUG logs Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/USE-373
ehanson8
approved these changes
Mar 12, 2026
ehanson8
left a comment
ehanson8
left a comment
Contributor
There was a problem hiding this comment.
Smart fix to tame the logs!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose and background context
This PR dramatically reduces
DEBUGlogging for TIM for bulk updates.DEBUGlogging is very helpful sometimes, potentially ever for large jobs, and much of theopensearchlogging is very helpful and informative. It does have one very chatty logging statement that is problematic:opensearch.base._log_request_response(). This logs both the API request + response. For bulk indexing and updating we are sending tons of data -- the TIMDEX records to index/update -- and they are all logged when inDEBUGmode.This may seem inconsequential for a few hundred records, but as we get into larger runs (thousands, millions?), the amount of data logged >= the amount of data we sent!
This PR filters out only this specific method from logging. If we need to know what data we send or receieve from an API request, we can do so; this
opensearchpylibrary logging line feels superfluous.How can a reviewer manually see the effects of these changes?
Easier to just share logs before after the change.
Here is a snippet pre-changes, showing how each record we send for update is getting logged:
Here is the same section of logs, missing this particular private method logging:
Includes new or updated dependencies?
YES | NO
Changes expectations for external applications?
YES | NO
What are the relevant tickets?
Code review